Optimize series key parsing on startup #6743
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
LGTM 👍 |
|
We just moved our server (~2T of InfluxDB data) to the nightly from last night (with this in it) vs the nightly from the 23rd May. The startup time goes from over an hour to just over 10 minutes. The system load during the startup is roughly doubled (though there are still boat loads of CPUs doing nothing). The output of "perf top" shows totally different limiting factors. Major CPU users now: This is a huge win. The next win is to just do more of startup in different goroutines, but thats a totally different kettle of fish. Thank you!! |
This was referenced May 31, 2016
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Required for all non-trivial PRs
This optimizes the
ParseKeyfunc to help speed up startup times for shards with many keys or very long keys. This should help #6250 in some cases.ParseKeyis called for every key when reloading the WAL and TSM file indexes. The main change is avoiding calling the unescape funcs for tag keys and values which always returned a new string even if there was nothing escaped. We also quickly scan the key to see if we might need to unescape anything and skip all of those calls if we don't need to. The remaining allocation time is creating the tags map. The tags map is really just a temporary holder to allow the tags to be indexed. We could rework this further to avoid that map altogether and parse the key directly into the index structures.